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PATTERNS IN THE MASS COMMUNICATIONS 
TASTES OF THE YOUNG AUDIENCE* 


PAUL I. LYNESSt 
Development Division, The Curtis Publishing Company, Philadelphia 


This article summarizes a recent study of the mass communi- 
cations habits and tastes of young people in school. The major 
goal of the study was to discover what ‘patterns of interest’ in 
the content of the mass media as a whole characterized five 
different age groups of boys and girls. By ‘patterns of interest’ 
we mean here the tendency for certain subject matter to cluster 
in the content preferences of any given age-sex group, and the 
tendency for these preferences to carry over from medium to 
medium.! 





* The study of which this article is a summary was the author’s doctoral 
dissertation in mass communications, written under the direction of Pro- 
fessor Leslie G. Moeller, Director of the School of Journalism, State Uni- 
versity of Iowa. 

t Formerly head of the Research Division, School of Journalism, State 
University of Iowa. 

‘ Alice Sterner (Radio, Motion Picture and Reading Interests; A Study of 
High-School Pupils, Contributions to Education, No. 932, Teachers College, 
Columbia University, 1947) has provided us with one approach to this 
problem of communication interest patterns. She strove to answer these 
questions: ‘‘What really impels youth to spend so much time (in mass 
communications activities?) . . . Is it the medium itself or a major ado- 
lescent interest in adventure, humor or love, which attracts the pupil?” 
Some other researchers who have studied interests in all or nearly all the 
media simultaneously are: Wilbur Schramm, ‘“‘Reading and Listening 
Patterns of American University Students,” Journ. Quarterly, March, 1945; 
Walter J. Scott, Reading, Film and Radio Tastes of High-School Boys and 
Girls, Wellington, New Zealand, 1947; Sidney L. Pressey and Francis P. 
Robinson, Psychology and the New Education, New York, 1944, pp. 126-139; 
Frederick J. Meine, ‘‘Radio and Press among Young People,” in Lazarsfeld 
and Stanton, Radio Research-1941, New York, 1941, pp. 189-223; Paul A. 
Witty and Anne Coomer, “‘Activities and Preferences of a Secondary- 
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It was felt that if rather detailed patterns of mass communica- 
tions interest could be established and described meaningfully, 
editors, station managers, and producers would find them useful 
in ‘beaming’ their output to specific audiences of young people. 
Such information would also enable teachers better to understand 
their pupils and to utilize the mass media more effectively in 
their teaching programs. Social researchers, too, preoccupied 
with the prediction and control of behavior, might find the 
material valuable in testing the interaction between youthful 
mass communications interests and personality, adjustment and 
behavior. 

With this as the problem, then, a rather comprehensive study 
was designed and carried out in the public schools of Des Moines, 
Iowa, in March, 1950.? 


METHOD 


Samples in five grades—third, fifth, seventh, ninth and 
eleventh—were selected by essentially random methods, although 
an initial stratification by sex and school attended was made. 
Students above the third grade filled out questionnaires in the 
classroom. Third-grade students were interviewed individually. 
The questionnaires were administered by teachers in the Des 
Moines public school system and graduate students at the State 
University of Iowa, all of whom were carefully briefed on the 
purposes of the survey and the procedures to be followed. The 
questions covered the amount and kind of reading, radio listen- 
ing and movie-going; the preferred types of mass communica- 
tions content (by medium); the names of preferred newspaper 
columnists, radio programs, movies, magazines, books and comic 
books; and preferred media. 

A series of six pretests involving pupils in the third, fourth, 
fifth, seventh, eleventh and twelfth grades permitted a practical 
evaluation of the study design, the questionnaires and the 
methods employed. 





School Group,” Journal of Educational Psychology, February, 1943, pp. 
65-76. ; 

2 The study was in large part financed by the Des Moines Register and 
Tribune Company. The writer wishes to thank Henry Kroeger, Director 
of Research and The Iowa Poll, Des Moines Register and Tribune, for his 
extensive personal assistance in carrying out the project. 
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CHARACTERISTICS OF THE SAMPLES 


The total sample of 1,418 school children was made up of 
six hundred ninety-one boys and seven hundred twenty-seven 
girls. Over three hundred students in each of grades V, VII, 
IX and XI took part in the study. In addition, one hundred 
fifteen third-grade students, fifty-nine boys and fifty-six girls, 
were interviewed. 

The breakdown of the total, by occupation of father, showed 
the following distribution: professional, eleven per cent; managers 
and proprietors, fifteen per cent; clerical and sales, twenty-two 
per cent; service, six per cent; skilled labor, twenty-two per cent; 
semi-skilled labor, eighteen per cent; unskilled labor, three per 
cent; other, three per cent. 

By education of father, the percentage breakdown for fifth- 
grade students was: grade school, 5.6; high school, 26.6; college, 
35.6; don’t know, 32.2. For seventh-grade students, the break- 
down in percentage (following the same order as above) was 4.6, 
47.4, 26.1 and 21.9 respectively; and for the ninth- and eleventh- 
grade respondents, approximately 13.5, 44.3, 28.5, and 13.6. 
Information on the education of parents was not obtained from 
third-grade students. 

Other background data obtained in the survey included owner- 
ship of automobiles, telephones, radios and TV sets; church 
affiliation and attendance; years of residence in Des Moines; 
preferred presidential candidate in 1948; possession of a library 
card; and recreational activities at home and away from home. 


REPRESENTATIVES OF THE SAMPLES 


The samples were substantially representative of their respec- 
tive grade populations, in terms of sex and school attended, by 
virtue of the initial stratification. Figures on parental education 
and occupation for all public school children were not available. 
However, an estimate of ‘representativeness’ in terms of these 
factors was made as follows: A statistical comparison was made 
of educational and occupational data gathered by The Iowa Poll’ 
on city adults with children in school and city adults without 





* The Iowa Poll is a weekly, state-wide survey of what Iowans think on 
leading topics of the day. It is sponsored by The Des Moines Register 
and Tribune. 
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children in school (state-wide surveys). The differences between 
these groups were insignificant at the five per cent level of 
confidence. 

Then, the educational and occupational data in the present 
survey were compared with three recent Jowa Polls of adults in 
the city of Des Moines. This test revealed no significant differ- 
ences in the occupational categories (at the five per cex:t level of 
confidence), except in ‘Managers and Proprietors,’ where the 
difference was significant at the two per cent level. The present 
samples contained more people from middle and lower status 
homes than The Jowa Poll samples. On the other hand, the 
test showed that the present samples were somewhat overloaded 
with children whose parents had attended high school and 


college. 


THE SURVEY CITY 


Des Moines, at the time of the survey, had a population of 
approximately 176,954 (1950 census). The morning Register 
and the evening 7'ribune, both published by the Des Moines 
Register and Tribune Company, were the city’s daily newspapers. 

All four major networks had outlets in Des Moines, and, in 
addition, two independent stations were located there. In 
1949, there were twenty-four moving picture theatres in the 
city. Des Moines citizens had access to over half a million 
library books in the ten public libraries, the Drake University 
library and the State Traveling Library. 


THE QUESTIONS RELATING TO PREFERRED TYPES OF CONTENT 


In the questionnaire given to fifth-, seventh-, ninth- and 
eleventh-grade pupils, seven questions dealt with preferred types 
of subject matter in the various communications media. One 
question was devoted to each of the following: newspapers, radio, 
moving pictures, types of magazines, magazine stories and 
features, books, and types of news. 

A check list of content categories accompanied each question. 
The pupils were asked to select and rank their four favorite 
types of content from each list. Thus, the question relating to 
motion pictures was phrased as follows: ‘‘11. Here are several 
different types of movies. Which four do you usually like best? 
Write ‘1’ before the type of movie you like best. Write ‘2’ 
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before your second choice. Write ‘3’ and ‘4’ before your third 
and fourth choices.” 

The check list of motion picture categories appeared below 
the question (See Table III). 

The radio and magazine check lists contained several names of 
programs and magazines exemplifying the categories. The 
illustrative programs had high Hooperatings in Des Moines at 
the time of the study and represented the various networks and 
stations heard in Des Moines. For example, ‘‘Western shows 
like The Lone Ranger, Roy Rogers, Tom Mix, Hopalong Cassidy, 
etc.,” and ‘‘ Radio columnists like Jimmy Fidler, Louella Parsons, 
Walter Winchell, Sheila Graham, etc.”” The magazines named 
were high circulation magazines in each classification. For 
example, “Sports magazines like Sports Afield, Outdoor Life, 
Sport, etc.,” and ‘‘Girls’ magazines like Senior Prom, The Ameri- 
can Girl, etc.”’ 

In determining the preferences for each sex-grade group, the 
first, second, third and fourth choices for each medium were 
weighted and combined. (First choices were weighted 4; second 
choices, 3; third choices, 2; and fourth choices, 1.) The weighted 
scores for each medium were then converted to percentages. 
This made easier the comparison of preferences in the different 
media. 


FINDINGS 


Tables I-VII contain the content ratings for each sex-grade 
group, and for all media, plus types of news. 

It should be borne in mind that these ratings are based upon 
the weighted first, second, third and fourth choices, and not 
upon a single favorite ineach medium. The four-choice approach 
was used in order to get some measure of intensity or depth of 
interest. 

A tabulation of first choices alone disclosed high group interest 
in a few categories. For example, the number one newspaper 
favorites of about three-fourths of all respondents in all grades 
were either comics or sports. With only a quarter of the first 
choices remaining, to be distributed among fifteen to twenty 
categories, the ranking of the secondary interests became some- 
what difficult and imprecise. On the other hand, the tabulation 
of four weighted choices revealed strong secondary interests in 
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TaBLE I.—FavoriTeE Parts OF THE NEWSPAPER 
(In Per cents of Weighted Ist, 2nd, 3rd and 4th Choices) 

















Boys Girls 
5th | 7th | 9th |11th 5th | 7th | 9th |11th 
2.2) 2.1) 1.4) 2.0) Advertisements .8] 1.9) 1.0) 1.2 
ee a Book Reviews ie Me ee 
.6| .7| .3} .3)} Business pages | .4 1 
1.4} 1.1) 2.0) 2.9} Columns on public af-| .9} .9) 2.1) 2.8 
fairs 
5} .2} .8 .9| Columns on love &| 3.1) 4.8) 5.7/ 4.8 
personal problems 
33 .5/30.4/24 .7/24 .8) Comics 34 .4/34 .7|26 .8)/22 .9 
4.8) 3.2) 1.0) 1.1} Continued stories 5.3} 4.0) 1.1) .3 
1.4) .5) 2.0) 2.2) Editorials .6] 1.2) 1.0) 2.4 
1.2} 1.1) 1.4} 1.1} Gossip Col’s 1.0} 2.0) 3.4) 3.0 
6.2) 6.7| 7.8) 5.8) Headlines 8.0) 5.9) 7.1) 6.0 
.2| .3) .5| .4| Letters to editor Me oe: ae BS 
11.1|10.0) 7.8) 4.3) Movies or radio news [12.2/13.2) 8.8) 5.8 
4.7| 9.5|12.6)15.0| News (general) 7.6) 6.0)10.6/10.8 
8.6) 7.2) 7.8) 8.0} Pictures 12.1] 9.8) 7.5) 5.7 
1 Society; Women’s fea-| 1.6) 3.0) 5.6/12.2 
tures 
22 .7|26 .4/28 . 1/30. 5] Sports 9 .8)10.7|17 .0/20.3 
7] .2| 1.4] .7| Other - ae |S ae 
1 1 None a 
1456/1616/1547/1410) Weighted total 1557|1945/ 1458/1576 





























such newspaper categories as movie and radio news, women’s 
features, and general news, depending upon grade and sex. 
Comparisons of first choices with the combined four choices did 
show considerable correspondence in the direction and magni- 
tude of changes from grade to grade. 

One important qualification to the findings contained in 
Tables I-VII should be considered. The rating of each content 
category for each sex-grade group is a relative rating; that is, it is 
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relative to all other categories for a given medium. Thus, the 
fact that interest in newspaper comics fell off somewhat with age, 
relative to other newspaper categories, does not mean that the 
older children’s absolute interest in comics was less. It merely 


TaBLe II.—Favorite Types or Rapio PrRoGRAMS 
(In Per cents of Weighted Ist, 2nd, 3rd and 4th Choices) 

















Boys Girls 
5th | 7th | 9th |11th 5th | 7th | 9th |11th 
14.2)11.2) 4.3) 1.7) Adventure serials 9.4) 6.5) 2.9) 1.1 
13 .7/15.1/19.1|18.1| Comedy shows 13 .5/15.4/16.7|13.3 
1} .4| 1.1] 2.3) Classical music 1.2} 1.1] 2.1] 5.0 
.3| .5| .1)  .1) Love serials 3.6] 1.8) 1.8) .8 
12.118 .0/17 .3|17.4| Mystery plays 12.018 .2)16.7)11.7 
1.0) 2.8) 2.6) 3.4) News 1.3} .6) 2.0) 2.6 
8.5) 4.0) 5.3} 3.8) Oldtime and west. 7.7) 5.8) 2.9) 2.6 
music 
6.0} 7.5) 9.6/10.0) Other plays (radio 12.0)18. 7/20 .3/21.5 
drama) 
.5| 2.5) 6.8/15.8) Popular music 3.2) 7.8)15.1/23.3 
8.4| 7.6) 8.3) 4.2) Quiz shows 12.3)10.4| 7.9) 5.5 
.6| .7| 1.1) 1.5} Radio columnists .5| .6) 1.4] 1.8 
.4| .6) 1.1| 1.5) Religious programs 1.2} 1.9) 2.5) 2.1 
8 .9/12.4/13.7|15.5) Sports 2.7| 3.0) 4.1] 6.5 
.2;|  .1) .3| .2| Talks; Forums 3 3 
24 .8)16.2) 9.0) 3.6) Western shows 19.2) 7.4) 3.1) 1.1 
.3| .4, .3] = .9) Other , Sot wok ae 
1573/1632|1574|1379| Weighted total 1581/1930)1470/ 1587 





























means that the relative rank of comics among other newspaper 
categories was lower among the older boys and girls. 

Several questions in the study were designed to get from the 
boys and girls information on what they actually read, heard on 
the radio and saw in the movies. The responses to these ques- 
tions indicated a sizeable increase in the variety of reading, 
listening and viewing from the third through the eleventh grades. 
For instance, analysis of ‘regularly read’ newspaper content 
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showed an increased or stable proportion of responses, from grade 
to grade, in every category except ‘book reviews’ and ‘continued 
stories.’ Coming back to the example above, between 95 and 
100 per cent of the boys and girls in all grades said they read the 
comics regularly. 


TaBLeE III.—Favorite Types or Motion Pictures 
(In Per cents of Weighted Ist, 2nd, 3rd and 4th Choices) 

















Boys Girls 
5th | 7th | 9th |11th 5th | 7th | 9th aa 
11.1/14.8)18.4/19.6| Adventure 9.2) 9.6) 9.8) 9.4 
14.4) 8.0] 5.4) 3.7) Cartoons 16.8) 6.9) 3.0) 1.9 
14.7/15.5)13 .5)15.2} Comedy features 14.5)13 .8)12.7) 9.7 
2.1} 4.3) 4.1] 6.2) Historical movies .7| = .7} 3.0) 5.0 
.7| 4.9) 5.1) 2.7| Horror movies 1.0) 3.7) 3.2) 2.0 
1.3} 1.0} 2.2) 2.3) Love and romance 8 .0)12.9)12.8)14.8 
2.2) 1.7) 6.9)11.7| Musicals 11.7|18.1/22.2/28 .4 
9.110 .3)12.4/11.5) Mystery 9.5)10.3)13.5) 8.2 
.4| 1.0) 2.2) 1.2) Religious movies 1.9) 3.2) 4.9) 4.3 
.3| .3) .8] 1.5) Social problems 8} .7] 1.2) 3.5 
14.8/17.5|15.5)/13.7| War 1.8) 2.6) 3.6) 4.5 
28 .2)18.1)12.2} 8.9} Westerns 18 .6)11.8) 5.9) 2.5 
.7| 1.8} 1.2) 1.4) World of make believe | 4.4) 4.5) 3.0) 3.4 
.8| .1) .4| Other drama 1.1) 1.2) 1.2) 2.4 
1339/1567|1559/1340) Weighted total 1365)1877/1457/1519 





























In radio program types listened to regularly, there was an 
increased or stable proportion of responses, from grade to grade, 
in all categories, except quiz shows, westerns, adventure serials, 
old-time and western music, and love serials. 

Keeping in mind these reservations, one may make three 
general statements with respect to the data contained in Tables 
I-VII. 1) Both grade and sex influenced the degree of prefer- 
ence for certain types of content. 2) Preferences for certain 
types of content tended to remain at about the same level from 
age group to age group. 3) The tables disclose at least three 
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hypothetical ‘patterns of interest’ or generalized interests, which 
may be described as (a) violence and adventure; (b) love, private 
life, and glamor; and (c) ‘educational’ content (content which 
may be described as primarily informative and serious). 


TaBLE IV.—Favorite Types oF MAGAZINES 
(In Per cents of Weighted Ist, 2nd, 3rd and 4th Choices) 




















Boys Girls 
5th | 7th | 9th |11th 5th | 7th | 9th |11th 
3.7| 3.8) 4.6) 1.3| Adventure 2.0) 1.6) .6 .1 
12.4:15.812.8) 6.2) Boys’ | oo oe Re, 
9.7| 8.8) 6.1) 3.7) Detective story 7.6; 5.3) 3.0) .4 
4.0) 2.7| 2.4) 1.4) Farm - ii ee wae 
2.6) 3.9,11.7|14.4| General appeal 4.4) 7.3)11.9)13.5 
. a oe Girls’ 11.4/16.6)15.7/15.1 
1.5) .9| .9| 1.8) Home service 6.2) 4.6) 3.3) 6.3 
4, .1) .1) Literary a 2 
4.1) 4.9) 5.6) 9.0) Men’s = 4... ef 
3.4) 3.6) 2.8) 1.6) Movie 12.6)16.4/14.3) 9.0 
1.1) 1.4) 1.3} 2.8) News i: oe ee | 
5.5) 7.8) 9.5,16.5) Picture 9.4) 7.9) 9.2/11.2 
8.1) 9.4)12.4/14.1) Science 19 6 .& .2 
19 .1|/20 .6/21 .7|23 .3| Sports 6.8) 3.2) 3.0) 2.3 
.8} 1.3) 1.5) 1.2) Religious 1 2} 1.8) 2.0) 1.2 
1.0) 1.1); 1.2} 1.0} Romance 8 .5)12.3)11.5) 7.2 
22 .0)13.0) 4.8) 1.6) Western 13.5) 5.8) 2.3) .3 
. a Women’s 8.2) 8.4) 9.1/10.3 
3} «2 Young women’s 2.1) 6.9)12.5/20.7 
1312)}1562/1560/1364) Weighted total 1354/1887|1441|1580 


























I, INFLUENCE OF GRADE AND SEX ON PREFERENCES FOR CERTAIN 
TYPES OF CONTENT 


Those effects of grade and sex which were especially marked 
will be discussed under two major headings: 

A) Categories which rose in relative popularity with the grade 
level of the respondents; 
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Both sexes.—The preference level of both boys and girls rose 
steadily, from the fifth through the eleventh grades, in the case of 
sports and general news in the newspaper; radio drama, popular 
music and sports on the radio; movie musicals and historical 
movies; general magazines; sports stories and features in maga- 
zines; and international and foreign news in types of news. 

Popular music, radio drama, and movie musicals were con- 
sistently more popular with the girls than the boys. Newspaper 
sports and general news, radio and magazine sports, and inter- 
national news were consistently more popular with boys than 
girls. 

Boys.—The boys’ preference level, but not the girls’, increased 
with respect to the following: movie adventure; science and 
invention in magazines, books, and types of news; men’s maga- 
zines; picture magazines; adventure books and books on sports. 
All these categories were more popular with boys than girls, 
except for picture magazines and adventure books which were 
about equally popular with the sexes. 

Girls.—The girls’ preference level, but not the boys’, increased 
with respect to the following: women’s features and society, and 
gossip columns in the newspaper; classical music on the radio; 
love and romance, religion, and war in the movies; love and 
romance, and women’s subjects in magazines; magazines for 
women and young women; historical fiction, and love and romance 
in books. All these categories, except war and historical fiction, 
were more popular with girls than boys. 

B) Categories which fell off in relative popularity as the 
grade level of the respondents rose: 

Both sexes.—The preference level of both boys and girls 
dropped, from the fifth through the eleventh grades, in the case 
of comics, movie and radio news in the newspaper; westerns in 
radio, movies, magazines and books; adventure serials, old-time 
and western music, and quiz shows on the radio; movie cartoons; 
detective story magazines; and auto wrecks and Hollywood in 
types of news. Westerns and adventure serials were consistently 
more popular with boys than girls. Movie and radio news was 
consistently more popular with the girls. 

Boys.—The boys’ preference level, but not the girls’, fell off 
with respect to boys’ magazines and movie magazines; and crime 
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TaBLE V.—FavoriTeE Types oF MAGAZINE STORIES AND 
FEATURES 
(In Per cents of Weighted Ist, 2nd, 3rd and 4th Choices) 























Boys Girls 
5th | 7th | 9th |11th 5th | 7th | 9th j11th 
15.2)13 .3/13 .6)17. 2} Adventure 12.4) 9.4) 8.6) 6.1 
2.2; 1.1) .5) 1.4) Art and music 6.1) 2.5) 2.3) 3.0 
.4| .3} .2| .6) Business and industry a ae 1 
6.1) 7.4) 7.8) 5.0) Cartoons 8.2) 8.6) 6.7) 5.4 
8.4) 8.1) 9.0) 6.7) Crime; Police; 4.9) 4.6) 4.7) 1.2 

gangsters 
5.4) 3.3) 4.5) 5.0) History and hist’l 1.2) 1.2) 2.1) 3.9 
fiction 

2.6| 2.2) 1.2) 1.8) Hollywood and movies | 8.9)10.7/11.3) 7.6 
44.1] .1| .4)| Home service 41.1) .1) 1.4 
10.1)11.8)12.5)10.6} Humor and comedy 13 .9]15.9/16.6)17 .3 
.3| .5| .2| Letters to editor Al 611.6] .7 
.8| .9| 2.0) .9| Love and romance 6 .6)13 .5/13 .9)18.8 
10.1) 9.8) 7.8} 8.0) Mystery; detective 8.2, 9.6; 9.0) 4.4 
.1] 1.0} 1.5) 1.4] Public affairs a 37 
.3} 1.2} 1.6} .9} Religion and churches | 2.7| 2.0) 2.9) 2.1 
2.8) 4.3) 8.4) 9.2) Science and invention | 1.3) 1.4! 1.7) 1.3 
.6| .5) 1.3) .4/ Social problems 1) .7| 2.4 
10 .5)16.1/17 .8|22.9| Sports 3.7| 4.1) 5.6) 8.9 
.2} 1.7] 1.4) 2.4! Travel 1.0} .7| 1.2) 2.7 
22 .4/13.9| 7.0) 4.1) Westerns 14.7) 6.1) 2.3) 1.2 
a Women’s subjects 1.6) 5.0) 7.0) 9.0 
1.4; 2.0} .4| .2) World of make believe | 3.7) 2.4) 1.5) 1.0 
.6| .9| .7| Other 3 2 
1242/1556/1519/1334) Weighted total 1359 1790) 1454/1557 
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in types of news. Boys’ magazines and crime news were more 
popular with boys than girls. Movie magazines were more 
popular with girls than boys. 


TABLE VI.—FavoritTe Tyres oF Books 
(In Per cents of Weighted Ist, 2nd, 3rd and 4th Choices) 



































Boys Girls 
5th | 7th | 9th |11th 5th | 7th | 9th |11th 
12.8/15.1/19.2)18.3) Adventure 14.6/16.2)14.9)15.8 
2.7; 1.1) .6) 1.5) Art or music 8.5) 3.0} 2.1] 3.2 
.8| .9} 2.2) 1.2) Biography 2.7| 1.4) 2.1) 3.9 
6.1) 4.1) 5.0) 5.5) History and hist’] 1.0) 2.0) 3.6) 9.2 
fiction 

.2| .4| .5| .1)| Home service .6} 1.0) 1.7] 1.4 
13 .4)11 .8)/12.0)13.0) Humor and comedy 14 .8/20 .4/21 .0/20.6 
.5| .7| 1.5} 1.2) Love and romance 8 .5)13 .6|15.6)18.1 
9.8)10.2) 8.8) 7.5) Mystery; detective 10.1/13.7|13.1) 5.9 

.2| .3| .8| .9] Public affairs . a aa 
.9| 1.0) 2.2) Religious books 2.1) 2.6) 3.7) 3.2 
3.9} 5.6) 8.3) 7.1] Science and invention | 3.0} 2.7) 1.6) 1.6 
2 .2 ol Social problems A om: 1.7 
13 .1/16.8/19 .2/22.0| Sports 7.1) 5.6) 8.5) 6.9 
.6; 1.8) 1.3) 1.9) Travel 1.9) 1.2) 1.9) 1.9 
11.1/12.2) 8.0)11.4) War 1.2} 1.2) .9) 1.5 
22.1/14.7| 8.3) 5.0) Westerns 17.3} 8.1; 4.0; 2.0 
2.1; 1.6) 1.3} .4| World of make believe | 6.1) 4.8) 2.6) 2.3 
4; 1.6) 1.9) .8) Other .o| 2.2) 1.8] .8 
1319}1595)1561/1391) Weighted total 1429) 1882/1489) 1566 











Girls.—The girls’ preference level, but not the boys’, fell off 
with respect to magazine adventure; the world of make believe in 
magazines and books; and sports magazines. Magazine adven- 
ture and sports magazines were more popular with boys than 
girls. The world of make believe was more popular with girls 
than boys. 
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TaBLE VII.—Favorite Types or NEws 
(In Per cents of Weighted Ist, 2nd, 3rd and 4th Choices) 









































Boys Girls 
5th | 7th | 9th |11th 5th | 7th | 9th |11th 
16.3|13.6| 8.9) 6.9) Auto wrecks; accidents | 9.3/13.4) 9.2) 6.4 
5.4) 1.9} 1.2) 1.6) Art, lit., and music 12.0) 5.4) 3.8) 5.9 
.1} .3} 1.1) 1.0) Business and industry — a on le 
18 .9/17 .3|16.3)13 .6) Crime; police; 10.8)11.1)12.6) 8.9 
gangsters 
8.1} 9.8) 8.6) 8.4; Des Moines news 13 .0)10 .5)12.9)14.5 
9.6) 7.9} 8.0) 9.1) Disasters 6.9) 9.7|)10.6) 9.2 
3.4) 7.3)10.4| 7.7| Internat’l and foreign | 3.9} 3.9] 7.0) 7.5 
news 
.5|  .7| 2.2) 1.5} Labor Prob. a ae 
1.2} .2| .7| .1| Love and marriage 5.9} 4.3) 3.4) 4.2 
problems 
1.6) 2.0) 3.3) 4.2) Military affairs nn 4 a aS 
6.4) 5.8) 3.0) 2.5) Movies and Hollywood |16.8/19.0/14.5) 9.6 
1.1; .6) 2.5) 3.3) Politics and gov’t .8)  .9) 1.2) 2.7 
1.1} 2.2) 1.6) 1.3} Religion and churches | 2.8) 3.7) 5.1) 3.0 
4.6) 5.9} 8.0)10.2) Science and invention | 3.1) 3.2) 1.8) 3.7 
.8} .7| 1.1) .3/ Social problems .9} 1.1) 1.6) 2.9 
20 . 6/23 . 6/22 .8/28 .3) Sports 12.7/12.2)14.5)19.1 
| a er Other .6| .3] .4 
1368) 1609/1577|1388| Weighted total 1438/1909 1480/1580 





II. CONTENT PREFERENCES WHICH TENDED TO REMAIN AT ABOUT 
THE SAME LEVEL FROM AGE GROUP TO AGE GROUP 


Those categories showing consistent ratings from grade to 
grade will be discussed in terms of both sexes, of boys, and of 
girls. 

Both sexes.—The following categories drew high and con- 
sistent ratings from boys and girls in all grades: newspaper 
comics and sports (although the percentage score for comics fell 
off somewhat while the score for sports rose); comedy in the 
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movies and on the radio; humor in magazines and books; mystery 
and detective in radio, movies and books. Sports and movie 
adventure were more popular with boys than girls. Humor in 
magazines and books was more popular with girls than boys. 

A consistent secondary interest was shown in horror movies; 
crime, cartoons, history and historical fiction and mystery in 
magazines; disaster, religion, and art-literature-music in types 
of news. 

Consistently low in the preferences of both sexes were adver- 
tisements, book reviews, public affairs, and gossip columns, 
in the newspaper; letters to the editor in newspapers and maga- 
zines; business and industry in all media; news, religious pro- 
grams, talks and columnists on the radio; social problems in all 
media; serious drama in the movies; farm, literary, and news 
magazines; travel articles in magazines; labor problems, military 
affairs, and politics in types of news. 

Boys.—The following categories rated consistently high with 
the boys in all grades: war in movies and books; adventure stories 
in magazines; and sports magazines. All were more popular with 
boys than girls. The boys showed a secondary interest in 
humorous books, adventure and movies magazines, and history 
in books. 

Low on the boys preferred lists were columns on love and 
personal problems in the newspaper; love and romance in movies, 
books, and magazines; fantasy, and religion in the movies; 
magazines for girls, women and young women; home service and 
romance magazines; art and music in books and magazines; 
Hollywood in magazines; biography in books; and news of love 
and marriage problems. 

Girls.—The following categories rated consistently high with 
the girls in all grades: romance, movie, and girls’ magazines; 
humorous books; magazine stories about the movies and Holly- 
wood; and crime news. All were more popular with girls than 
boys, except crime news. 

The girls showed a secondary interest in columns on love and 
personal problems in the newspaper; religion and fantasy in the 
movies; home service and picture magazines; books on sports, 
religion and art or music; biography; art and music in magazines; 
science and invention in books and types of news; and love and 
marriage problems in types of news. Historical movies; adven- 
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ture, science, boys’, men’s, and science magazines; and war 
books were low among the girls’ preferences at all grade levels. 


III. THREE HYPOTHETICAL PATTERNS OF INTEREST 


In order to get a more generalized picture of interest tendencies 
among young people, three hypothetical interest patterns were 
established, under which specific subject-matter categories 
could be classified. It was desired to uncover the extent to 
which these patterns characterized any given grade-sex group. 

The three hypothetical patterns were (a) Adventure and 
Violence; (b) Love, Private Life and Glamor; and (c) ‘Educa- 
tional’ content. All categories (in each medium) which were 
subsumable under these headings were grouped together for 
analysis. 

Mean preference ratings for each pattern were calculated for 
each grade-sex group by dividing the sum of the ratings for all 
applicable categories in each and every medium, by the total 
number of occurrences of the categories in each and every 
medium. There were twenty-two total occurrences in the adven- 
ture and violence classification; thirteen occurrences in the case 
of love, private life and glamor; and thirty-three occurrences in 
the case of ‘educational’ content. These ratings were subject 
to the characteristic disadvantage of means in that they could 
be skewed by extreme cases; however, there were few extreme 
cases in the distributions concerned. Figure I shows graphi- 
cally the mean ratings of the three hypothetical patterns of 
interest by grade and sex. 

The results indicated marked sex differences in the preferences 
for adventure and violence, and love, private life and glamor. 
The boys’ ratings exceeded the girls’ in every violence and 
adventure category. The girls’ ratings exceeded the boys’ in 
every love, private life and glamor category. Neither sex 
indicated any substantial preference for ‘educational’ content. 

There were also clearcut differences by grade in the case of 
adventure and violence, where the relative preference of both 
boys and girls declined steadily as the grade level rose. Differ- 
ences by grade in the preference shown love, private life and 
glamor were not clear cut. 

It should be emphasized that most of the individual categories 
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Ficure I. Comparison of Three Hypothetical Patterns of Interest, by 
Preference Level, Sex and Grade. 


making up each pattern occurred in more than one medium; for 
example, mystery occurred in radio, movies, magazines and 
books. Accordingly, the interests discussed in this section 
reflect subject-matter preferences in all media combined. 
Examination of individual categories revealed certain distinct 
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trends related to sex and grade differences. Within the adven- 
ture and violence pattern, interest in westerns dropped sharply 
with increasing age levels. Fifth-grade boys and girls gave 
westerns a far higher rating than any other category; yet eleventh- 
grade boys and girls gave westerns a relatively low rating. The 
same tendency could be observed in ratings of radio adventure 
serials. The ratings for mystery, crime, disaster, war, and 
adventure other than radio serials, remained at about the same 
level from grade to grade. Also, interest ratings for these five 
categories were roughly of the same magnitude within grades. 
These tendencies applied to both sexes, although in all cases boys 
gave higher ratings to adventure and violence than girls. Both 
sexes rated horror lowest among all categories classifiable as 
adventure and violence. 

All individual categories classified under love, private life and 
glamor were given relatively inconsequential ratings by boys. 
However, some interest in movies and Hollywood was shown by 
boys, especially in the fifth and seventh grades. Girls gave high 
ratings to love and romance, and movies and Hollywood. A 
tendency for the love and romance ratings to rise as the girls’ 
grade level advanced was noted Conversely, interest in movies 
and Hollywood appeared to drop progressively as the girls’ grade 
level advanced. Considerably less female interest was shown in 
the remaining categories classified under love, private life and 
glamor; that is, radio love serials, news on love and marriage 
problems, newspaper columns on love and personal problems, 
radio columnists, and newspaper gossip columnists. 

Eighteen individual categories were classified under ‘educa- 
tional’ content. With a few exceptions, these categories were 
given low ratings by both sexes. Relatively high ratings were 
given science and international and foreign news by ninth- and 
eleventh-grade boys. History elicited a moderate preference 
rating from boys, regardless of grade, and from eleventh-grade 
girls. Ninth- and eleventh-grade girls also ranked international 
and foreign news notably higher than other ‘educational’ 
content. Fifth-grade girls showed a substantial preference for 
art, literature and music. These exceptions of course, were not 
reflected in the mean ratings shown in Figure I, because of 
the generally low ratings of all other categories under ‘educa- 
tional’ content. 
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STRENGTH OF EACH PATTERN BY MEDIUM 


An indication was desired of the consistency of interest in the 
three hypothetical patterns from medium to medium. Accord- 
ingly, the preference ratings of all categories applying to each 
pattern within each medium were averaged. For example: In 
order to obtain the adventure and violence rating for radio, the 
preferences for the three pertinent radio categories were averaged; 
that is, adventure serials, mystery plays, and western shows. 
For movies, there were five pertinent categories—adventure, 
horror, mystery, war, and westerns. 

A considerable degree of consistency of interest from medium to 
medium was observed for certain grade-sex groups. Adventure 
and violence in all media appealed strongly to fifth- and seventh- 
grade boys. Love, private life and glamor in all media but 
radio appealed strongly to seventh- and ninth-grade girls. 
‘Educational’ content in all media, except magazines, received a 
low preference rating among boys of all ages. The somewhat 
higher male interest in ‘educational’ magazines, was due to the 
substantial popularity of science and mechanics. The girls’ 
ratings of ‘educational’ content were uniformly low in all media. 
On the other hand, marked media differences in the appeal of the 
hypothetical patterns to certain grade-sex groups were observed. 

Analysis of differences in the adventure and violence classifica- 
tion revealed that the older the student the less consistency from 
medium to medium. Adventure and violence in radio and 
magazines was much less popular among eleventh-grade boys 
than fifth-grade boys, although male interest in movie and book 
adventure and violence continued at a high level in all grades. 
Girls’ preferences for this pattern followed the same general 
tendency; that is, for media differences to develop in the upper 
grades. Little interest on the part of either sex was shown in 
adventure magazines. This may have been due to the relative 
vagueness of the category. 

Girls’ ratings of the love, private life and glamor categories 
revealed a tendency for magazine stories and features and movies 
to become more popular relative to the other media from the 
fifth through the eleventh grade. Conversely, the relative 
popularity, for girls, of magazines devoted entirely to love and 
glamor appeared to decrease sharply after reaching a peak in 
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the seventh grade. Radio categories in the love, private life 
and glamor area had no significant appeal for either sex at any 
grade level. 

Under love, private life and glamor, radio and books were the 
weakest media. The two radio categories were love serials and 
radio columnists. The book category was love and romance. 

The male preference for ‘‘educational”’ content in magazines, 
as against other media, was due in large part to an interest in 
magazines on scientific and mechanical subjects. 


THIRD GRADE PREFERENCES 


Questions used in the third-grade interviews were largely of the 
free answer type; no check lists were used. Hence, the results 
were analyzed separately from those obtained in the higher 
grades. 

Third-grade tastes, as revealed by this study, were focused 
mainly on the comics, westerns, animal themes, the world of 
make believe, adventure, comedy and humor, and mystery. 

By medium, the prevailing choices of both boys and girls were 
comics in the newspaper; westerns on the radio and in the movies; 
fairy tales and fantasy in books; adventure serials on the radio; 
comedy and humor in movies and magazines; animal themes in 
magazines and books, and mystery in radio fare, movies and 
magazines. 

Boys, more than girls, favored western movies and books; and 
mystery on the radio. Girls, more than boys, favored comedy 
and humor in movies and books; and animal themes in books. 

Third-graders indicated little preference for love, private life 
and glamor, although a considerable number expressed their 
liking for the movie and radio sections of the newspaper. They 
also show little interest in informative, non-entertaining content 
in any medium. 
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The Air University’s Air Command and Staff School, like other 
schools of higher learning, has been faced with the problem of 
evaluating various types of seminar performance not easily 
measured by objective tests. Among these is the problem of 
rating students’ performance in oral presentation. Several 
methods have: been tried for such ratings. New methods are 
being tried. The purpose of the present study is to evaluate the 
results of one general procedure used in teaching and rating 
oral expression at the Air Command and Staff School. This 
material is presented because the unique character of the course 
and rating system may be of benefit to those engaged in similar 
educational activity. 


DESCRIPTION OF THE COURSE 


The course in oral expression has been established to develop, 
in Air Force officers, the ability to organize speaking material 
and to present it logically and forcefully. This objective arises 
from the very practical needs of Air Force officers to speak before 
civilian organizations, to participate in Air Force conferences or 
joint conferences with representatives of the other services and 
the departments and agencies of the Federal or State govern- 
ments, to brief their commanding officers, and to participate 
in staff meetings. Because the length of the course is only fifteen 
class hours, the instruction is oriented around formal oral pres- 
entation on the assumption that speaking before large groups 
would be the most difficult type of requirement to be met in the 
field. Only the elementary fundamentals of oral expression are 
taught. Students are not expected to be finished conference 


members or speakers at the end of the course. 
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The instruction includes three one-hour lectures on essential 
steps in preparing a speech, divisions and organization of a 
speech, and effective speaker projection. The remaining class 
periods are devoted to presentations by students. Each student 
is required to make four oral presentations. The first require- 
ment is a five-minute talk on some phase of preparing a speech. 
The second requirement is a ten-minute talk on a subject of the 
student’s choice, but of military significance. The third require- 
ment is a five-minute impromptu presentation on the student’s 
choice of one of four subjects within the broad areas of opera- 
tions, personnel, logistics, and world events. This is a test of 
the student’s ability to delimit a broad subject and to speak 
spontaneously on that subject. The fourth requirement is a 
prepared fifteen-minute talk wherein the student regards himself 
as a numbered Air Force staff officer required to give a speech on 
some phase of the military establishment to a civilian audience. 

The total enrollment of four hundred sixty-five students is 
divided into small groups of about ten students each for the oral 
presentations. An instructor is assigned to each group. Both 
instructor and students rate each oral presentation on a rating 
scale described later in this report. All students are required 
to make the first three presentations. Those students in the 
top twenty per cent of the class are exempted from making the 
last presentation. Each rating provides a basis for grading the 
student and for constructive criticism. 


DESCRIPTION OF THE RATING SCALE 


The rating scale, devised by the Evaluation Branch of the Air 
University’s Educational Advisory Staff, was made up of twelve 
items rated on a scale of one to five. Items were selected on the 
basis of characteristics which, it was believed, could be observed 
and remembered during the time the speaker was on the platform 
and which could be used as a guide for the speaker while preparing 
his speech. The number of items was limited to reduce rating 
time as much as possible and still provide sufficient coverage of 
qualities making for good performance. 

These considerations were necessary since student-raters were 
being used and a number of ratings were made during each class 
period. The rating scale follows: 
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EVALUATION FORM FOR ORAL PRESENTATIONS 


I. Platform Presence.—Does speaker appear poised and purpose- 
ful? Is he well groomed? 
1 2 3 4 5 
Ill-at-ease and untidy Slightly self-con- Poised and immacu- 
appearance detracts scious; presentable; late; excellent plat- 
from platform presence. acceptable platform form presence. 
presence. 





II. Introduction.—Is introduction effective in getting attention, 
and audience acceptance? Is objective understood? Did 
he establish importance of subject and lead smoothly into it? 

1 2 3 4 5 

Satisfactory introduc- Complete and very 

tion, fairly complete; effective introduction. 

adequate but not ef- 

fective. 





Inadequate introduc- 
tion; failed to accom- 
plish one or more of the 
purposes of an intro- 
duction. 


III. Actions.—Do posture, body movements, and_ gestures 


increase effectiveness of presentation? 

1 2 3 4 5 
Fairly natural ac- Natural actions 
tions; annoying man- greatly aid presenta- 
nerisms slightly dis- tion; no distracting 





Unnatural, awkward 
actions; annoying man- 
nerisms definitely dis- 





tract from presenta-  tracting. mannerisms. 
tion. 
IV. Eye Contact.—Does speaker look at each member of the 
audience? 
1 2 3 + 5 
Eye contact detractive; Eye contact fairly ef- Eye contact very ef- 


fective, well distrib- 
uted, constantly 
looked at audience. 


tended not to look at fective; tended to 
audience. look at audience 


V. Voice.—Is voice pleasing? Can you easily and clearly hear 


him? Is he fluent and forceful? 
1 2 3 4 5 





Voice boring, irritating; 
indistinct, hesitant de- 
livery. 


Voice pleasing, fairly Voice very pleasing 
clear; matter of fact and clear; eloquent, 
delivery. stimulating delivery. 
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VI. Phraseology.—Does he pronounce and use words correctly? 
Are sentences short, clear and complete? Does he use 
terms clear to audience? 





l 2 3 4 5 
Annoying grammatical Minor grammatical Excellent grammar 
errors. Poor choice of errors. Satisfactory and sentence struc- 
words. Does not de- choice of words; ture; effective choice 
fine terms; difficult to meaning generally of words; easily 
follow. clear. understood. 


VII. Body.—Is plan of presentation clear to audience? Are 
major points effective and properly supported by minor 
points and details? 





l 2 3 4 5 
Plan not clear, too Plan fairly clear; Plan very clear; 
many or too few points fairly adequate major proper number of 
inadequately sup- points, moderately major points, well 
ported. supported. supported. 


VIII. Transition.—Does speech move smoothly and effectively 
from each part or point to following part or point? Does 
it have continuity? 





] 2 3 4 5 
Disjointed transition; Fair transition be- Smooth transition; 
parts not fitted to- tween parts; fair con- parts fit naturally to- 
gether; lacks contin- tinuity. gether; excellent con- 
uity. tinuity. 


IX. Knowledge of Subject—Did speaker have sufficient back- 
ground of facts on subject? Did he understand implica- 
tions of material? Were his conclusions sound? Did he 
adequately cover and emphasize important aspects of 





subject? 
l 2 3 4 5 

Lacked essential knowl- Fair knowledge of Knew subject thor- 
edge of subject and its subject; conclusions oughly; drew sound 
implication; conclu- generally adequate; conclusions. Effec- 
sions faulty; failed to partially covered im- tively covered impor- 
cover important as- portant aspects. tant aspects. 
pects. 


X. Conclusion.—Does conclusion effectively end speech? Does 
it have punch? Is it consistent with purpose? 
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1 2 3 4 5 


Weak, inadequate con- Fairly effective end- Concise, forceful end- 
clusion; new material ing, fairly consistent ing; drives home pur- 
introduced. Leaves with purpose; lacks pose. 

audience up in air, not punch. 

consistent with pur- 

pose. 





XI. Preparation.—Was speech extemporaneous (not memorized), 
and was it delivered without excessive use of props such as 
manuscript, notes or lectern? Was it within time limits 
and was proper time apportioned to developing each point 
or part? Were visual aids effectively used? 

1 2 3 4 5 

Indicated inadequate Indicated fair prepa- Indicated thorough 

preparation by exces- ration by moderate preparation by no re- 

sive use of props; poorly use of props; fairly liance on props; ex- 





apportioned timing good timing; accept- cellent timing and ef- 
with speech too long or able use of visualaids. fective use of visual 
too short; faulty use of aids. 

visual aids. 


XII. Interest—Did speaker establish and maintain audience 
interest in his subject and in himself as a speaker? Was 
audience reaction to speaker and subject friendly? 

1 2 3 4 5 

Dull and boring speak- Moderate audience Intense audience in- 

er, no audience interest; interest; audience re- terest; audience reac- 

audience reaction indif- action fairly cordial. tion very friendly. 
ferent or unfriendly. 





The grade for any one oral presentation was composed of the 
average of all students’ ratings on this form added to the instruc- 
tor’s rating. Ratings are corrected for rater bias by simply 
adding or subtracting the rater’s deviation from the mean of all 
ratings. Instructor ratings and students’ ratings are corrected 
independently. 

An indication of the reliability of ratings made on this scale 
was determined by finding the correlation between the average 
of student ratings| and instructor ratings on each student’s 
presentation in the second oral requirement (a ten-minute pre- 
pared speech on a subject of military significance). The cor- 
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relation between four hundred sixty-five scores paired in this 
way was .63 + .02. Although this correlation is not as high as 
is generally desired for reliability coefficients, it does represent a 
significant relationship of better than moderate magnitude. 

Further information about the rating scale is presented in 
Table I where intercorrelations of ratings on each characteristic 
with ratings on every other characteristic and the total score 
are shown. The item numbers in this table correspond to the 
item numbers in the rating scale. 


TABLE I.—INTERCORRELATIONS BETWEEN SCORES ON EACH OF 
THE TWELVE FACTORS OF THE EVALUATION ForRM FOR ORAL 
PRESENTATION AND THE TOTAL ScORE* 














(N = 465) 
Item Rating Scale Item Number* 
Num- Total 
bert |I| IT | IIIT} IV} Vj VI | VII| VIII; 1X} X | XI | XII 
I .46| .57| .47| .55| .48] .42| .61 | .44] .44] .35] .53|] .73 
II .83] .29| .37] .85] .44] .42 | .42] .40] .29] .48/] .62 
III .52| .42] .36] .30] .38 | .41] .31| .26] .42] .62 
IV .38| .34] .11] .41 | .35] .31] .26| .44 .61 
V .47)| .36| .42 | .38] .38] .383] .46| .65 
VI .42| .51 | .47] .45] .37| .46 .68 
VII .68 | .6&| .50| .50| .63) .75 
VIII .68| .50| .54/ .58| .79 
IX .50| .53| .62] .77 
x .56| .56|) .70 
XI .46| .65 
XII .78 












































* Data is from the instructor ratings on Oral Presentation III only. 
t Item numbers correspond to numbers opposite the rating scale items 
shown in the text. 


These intercorrelations indicate the existence of a positive 
relationship among all items. Relatively higher relationships 
exist between each item score and the total score. These 
relationships are desirable because they indicate that, in general, 
each item evidently makes some unique contribution to the total 
score. None of the items correlates so highly with the total 
score that it could replace the entire scale. Items VII, VIII, 
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IX, and XII have relatively higher intercorrelations than other 
items. This is to be expected, since they are also related in the 
practical situation. Each of these four factors is essential to 
preparation of the subject-matter content of the speech. These 
results have not been factorized, however, to determine the 
extent to which the assumed factors or other factors actually 


are present. 
RESULTS 


The effects of the oral presentation program are illustrated in 
the data obtained from the rating scales. 

Table II shows the mean scores and standard deviation for 
each of the oral presentations. The scores shown are a com- 
posite of the average of nine student ratings plus one instructor 
rating. The total possible score is thereby 120. From this 
table it will be recognized that the rating scale reflected an 
increase in scores as the course progressed. Differences in mean 
scores between any one speech and each of the other speeches 
are significant at the one per cent level. 


TABLE IJ.—MEANS AND STANDARD DEVIATIONS OF THE ORAL 
PRESENTATION SCORES* 


Speech 
Number Number Mean SD 
I 460 85.8 10.75 
II ) 460 90.9 | 9.45 
III 460 89.1 9.82 
IV 360 92.5 8.16 


* Differences between means on Oral Presentations I and II, I and IV, 
II and III, and III and IV are significant. 


The lower mean score for Speech III may be interpreted as 
evidence of the validity of the scale. Speech III is the five- 
minute impromptu speech with only ten-minute preparation. 
On this basis it would be expected that the individual’s work 
would be poorer than on a prepared presentation. This inter- 
pretation is reflected in the lower mean score for the third speech 
than for the second or fourth speech. 

Intercorrelations between each speech and every other speech 
are shown in Table III. These correlations show a substantial 
relationship between student performance early in the course and 
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later performance. Lower correlations between scores on 
Speech IV and scores on other speeches are undoubtedly a result 
of the increased homogeneity of the class (the top twenty per 
cent on three earlier speeches were excused from making the 
fourth speech). 


TABLE III.—INTERCORRELATIONS BETWEEN EACH OF THE ORAL 
PRESENTATIONS AND FINAL ORAL PRESENTATION SCORES 
Crass 49-B, AC&SS 











Oral Presentation 
Final 
Orel S 
Presentation I II III IV N pap 
N = 460/N = 460|\N = 460\N = 360) 
I . 67 .58 .40 .85 
II . 69 .49 .89 
III .42 .84 
IV ‘va 
Final Score 




















A study was made of the progress of those individuals in the 
highest ten per cent of the class (N = 46) and those in the lowest 
ten per cent of the class (N = 46) selected on the first presenta- 
tion. The averages for these groups are shown in Table IV. 
The significance of the differences between the scores for each 
speech for each of these groups is shown in Table V. 

Although there is a drop of five points from average of Speech I 
to Speech III for the highest ten per cent, the difference is 
significant at slightly more than the five per cent level of con- 


TABLE IV.—MEANS OF ScorES FOR EAcH ORAL PRESENTATION 
MADE BY THE HIGHEST AND LowEstT 10 PER CENT orf STUDENTS 
ON ORAL PRESENTATION No. I 


Highest 10% Lowest 10% 

Speech on Speech I on Speech I 
I 105.2 67.4 
II 101.3 79.2 
III 99.9 80.8 


IV 86.7 
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fidence. None of the other differences in speech scores for this 
group is significant. 

The lowest ten per cent of the class on Oral Presentation I 
made gains significant at least at the two per cent level of con- 
fidence except for the progress shown from Oral Presentation II 
to III. Again, this is probably accounted for by the fact that 
Oral Presentation III was an impromptu requirement. 


TABLE V.—SIGNIFICANCE OF DIFFERENCES BETWEEN ORAL 
PRESENTATION AVERAGES MADE BY THE HIGHEST AND 
Lowest 10 Per CENT OF THE STUDENTS SELECTED 
ON THE Basis OF SPEECH I ScorREsS 
(N = 46 for each group) 











; Highest 10% Lowest 10% 
Differences 
Between Speeches: DF! x P DF| x P 

1 and 2 2 |5.21| >.05] 2 | 13.21] <.01 
1 and 3 2 13.28] <.20/ 2 19.20 | <.001 
1 and 4 1 32.76 | <.001 
2 and 3 2 .07 .98 2 1.66 | <.50 
2 and 4 2 14.28 | <.001 
3 and 4 3 9.63 | >.02 























RELATIONSHIP BETWEEN PERFORMANCE IN ORAL EXPRESSION AND 
OTHER FACTORS 


Correlations were found between the final oral expression 
grades and several other factors on which scores were available 
(see Table VI). The data studied were as follows: General 
Survey Test—a two-part test yielding a score on student achieve- 
ment in Air Force background subject matter, another score for 
achievement in mathematics, science, and English and a total 
score; the American Council on Education Psychological Examina- 
tion’s Q, L, and total score; the student officer’s rank; the student 
officer’s period of active service; the student officer’s age; con- 
solidated scores of fifteen class examinations; and grades on the 
final school problem, which is a simulated problem situation 
solved by a staff of sixteen student officers. 
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TaBLE VI.—CoRRELATION OF FINAL OrAL EXPRESSION GRADES 
AND TWELVE Factors 


(N = 465) 
Correlation with Oral 
Factor Expression Grades 
General Survey Test 
Military subjects 31 
Basic . 26 
Total .33 
ACE Psychological Examination 
Q score (quantitative) .22 
L score (language) . 30 
Total . 29 
Rank .00 
Months active service .10 
Age .06 
Class examinations (15 consolidated 
scores) 47 
Final school problem (staff problem) . 22 
Final class standing (including oral 
expression) .50 


From these correlations, very little relationship is shown to 
exist between oral expression and rank, age, and months of active 
service. The oral expression grades have a higher relationship 
with general intelligence, as measured by the ACE Psychological 
Examination (r = .30), staff action (r = .22), and military 
achievement scores (r = .31). A still higher relationship is 
noted between oral expression and classroom achievement on 
written examinations (r = .47). It is interesting to note that 
the oral expression grades correlate more highly with the Ameri- 
can Council on Education Psychological Examination’s L score 
(r = .30) than with the Q score (r = .22). The difference 
between these correlations is, via Fisher’s Z transformation, 
significant at the one per cent level of confidence. 


SUMMARY 


The oral expression course at the Air University is established 
for the purpose of imparting basic principles in the preparation 
and delivery of speeches. Practice is given through four speech 
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requirements. Appraisal of student performance for purposes 
of evaluation and guidance is made by student and instructor 
ratings of each speech. 

Results of the analysis of scores on the evaluation form for oral 
presentation indicate that (1) raters are differentiating between 
presentations; (2) growth and development in oral presentation 
takes place throughout the course; (3) no progress in oral pres- 
entation was reflected in the evaluation scale for those already 
qualified at this level of oral expression; (4) those who were 
originally very poorly qualified in oral expression developed to a 
point where their average score on the final speech requirement 
exceeded the entire class average on the first oral presentation. 
One might conclude that, as a group, the Air University students 
improve in oral expression throughout the period of instruction 
and, furthermore, that most improvement is made by those 
individuals who were poorest in oral expression before attendance 


at the Air University. 














THE EFFECT OF ATTENTION-GAINING DEVICES 
ON FILM-MEDIATED LEARNING* 


D. MORGAN NEU 
The Pennsylvania State College 


INTRODUCTION 


Film-makers have employed a wide variety of techniques to 
achieve emphasis and increase the teaching effectiveness of their 
products. In general, the devices employed to direct attention 
towards particular points of the content presented may be divided 
into two broad classes: ‘relevant devices’ and ‘irrelevant devices.’ 
‘Relevant devices’ include any emphasis technique that is 
related to, or operates upon, the specific point to be emphasized. 
For example, an ultra-close-up of part of a gun brings attention 
specifically and forcefully to that part of the gun. ‘Irrelevant 
devices’ include any emphasis techniques that call attention to 
the screen but are otherwise unrelated to the film content. For 
example, interrupting a film on a technical subject with a shot 
of a bathing beauty is an instance of an ‘irrelevant device.’ 

While current learning theories and the experiments that sup- 
port them are not entirely in accord on the relative effectiveness 
of these two methods of ensuring the perception and learning of 
the salient points in a lesson, the consensus seems to be that, in 
the long run, ‘relevant devices’ should -facilitate recall, while 
‘irrelevant devices’ should act as distractors and inhibit recall. 
This inference is obviously at variance with the results hoped for 
by producers who insert sequences that attract attention to the 
screen without relating to a specific point of information. It is 
important, therefore, to determine experimentally the effects on 
learning of such different attention gaining techniques. 


STATEMENT OF THE PROBLEM AND HYPOTHESES 


The major purpose of this study was to determine the relative 
effectiveness of three levels of attention-gaining devices; (1) 





*Summary of a dissertation completed in pursuance of the requirements 
for the Ph.D. degree at The Pennsylvania State College, 1950. The work 
described here was accomplished under the Instructional Film Research 
Program under Contract N6 onr-269, Task order VII, with the Special 
Devices Center, Office of Naval Research. 
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relevant devices, (2) no devices at all, (3) irrelevant devices. 
Since it is possible to insert devices in the sound track or the 
visuals, the réle of the medium of presentation has also been 
studied. In order to maintain a clear cut comparison between 
the sound and visual media, sound and visual attention devices 
were each studied separately. Three hypotheses were formu- 
lated for testing as follows: 

1) Device relevancy.—The hypothesis is that film-mediated 
learning will be facilitated by the use of relevant attention- 
gaining filmic devices, and inhibited by irrelevant attention- 
gaining devices. The hypothesis was tested by comparing 
versions of a film containing, respectively, relevant devices, no 
devices, and irrelevant devices. 

2) Device medium.—The hypothesis is that film-mediated 
informational learning will be equally facilitated by visual and 
sound devices of the same relevance. The hypothesis was 
tested by comparing learning from visual device versions with 
learning from sound device versions. This comparison is made 
separately for relevant and irrelevant devices. 

3) Device recall.—The hypothesis is that recall of the devices 
themselves is relatively independent of learning of the factual 
information content of the film. 


EXPERIMENTAL DESIGN AND PROCEDURES 


The Experimental Films.—To accomplish the comparisons 
required by the hypothesis, five versions of a film presenting an 
introduction to machine shop measuring instruments were pro- 
duced. The five versions included the following: 

1) Basic (identified as B in the tables): contains no experi- 
mental emphasis devices, but gives a clear, straightforward 
treatment of the subject. 

2) Visual relevant (VR): contains attention-gaining devices 
in the visuals that are related to the points of content being 
emphasized. 

3) Visual irrelevant (VI): contains attention-gaining devices 
in the visuals that are unrelated to the points of content being 
emphasized. 

4) Sound relevant (SR): contains attention-gaining devices in 
the sound track that are related to the points being emphasized. 
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5) Sound irrelevant (SI): contains attention-gaining devices in 
the sound track that are unrelated to the points being emphasized. 

Choice, number, and placement of the devices.—To ensure satis- 
factory comparability among the four ‘device’ versions, a panel 
of judges selected four sets of devices considered to be comparable 
in intensity and probable effectiveness in gaining attention. 

It was decided that the devices selected should not add infor- 
mation to the film, but should be used solely for their possible 
attention-gaining value. Furthermore, it was decided to select 
attention-gaining devices that would represent various categories 
of stimulus conditions: movement and change, size, intensity, 
repetitive or prolonged contact, novelty and peculiarity, impres- 
siveness, and sudden contrast. 

A device was inserted at each of the same twenty-six most 
important points in each version. Table 1 lists the devices and 
their placement in relation to the informational content of the 
film. Thus, a point emphasized in one version by a visual 
relevant device (e.g., a spotlight) is also emphasized in the other 
three versions, by appropriate devices. It will be noted that 
some devices are used more than once. 


THE TESTS 


Two tests were constructed to provide measures of (1) learning 
of information and (2) recall of devices. The information test 
comprised one hundred four multiple-choice items, of which 
twenty-five provided picture-choices rather than verbal choices. 
This test had a high reliability (.94 by Kuder-Richardson 
Formula 20). 

The device recall test included seventy-seven items, each of 
which briefly identified a device by a short statement. The 
subjects were called upon to check whether (1) the item was in 
the film, (2) the item was not in the film, or (3) they did not 
remember whether it was in or not. 


THE POPULATIONS 


The experiment was conducted with two separate military 
populations: (1) 1576 Army recruits at Fort Dix, New Jersey, 
(2) 1055 Navy recruits at The Great Lakes Naval Training 
Station, Great Lakes, Illinois. 
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PROCEDURES 


At each testing center, the following procedures were employed: 
intact companies of recruits were divided at random into six 
comparable groups. Five of these groups were each shown one 
of the film versions (B, VR, VI, SR, or SI) and, immediately after 
the film, given the information tests and the device recall test. 
The sixth (control) group was given the information test without 
having seen the film. Test responses were recorded on IBM 
answer sheets, and machine-scored. 


STATISTICAL TREATMENT OF THE DATA 


For the men in each population both a measure of general 
intelligence and a measure of mechanical aptitude were available. 
Mean scores for the information test for the six treatments were 
calculated, and these means were adjusted, by an analysis of 
co-variance, to take into account small differences in initial 
ability as measured by the general intelligence and mechanical 
aptitude tests. The adjusted means therefore reflect the per- 
formance to be expected of groups having equal mean general 
intelligence scores and equal mean mechanical aptitude scores. 

To determine the relationship between the information test 
and the device test, scores on these two tests were correlated. 


RESULTS 
Comparison of the Film Versions 


Information Test Results —In Table 2 the mean scores on the 
information test are given for the six groups of enlisted men in 
the Army and in the Navy. The adjusted means are given in 
parentheses. Table 3 summarizes the differences among the 
adjusted information test mean scores. 

The findings can be summarized for both the Army and Navy 
populations as follows: 

1) For all film groups substantial learning resulted from viewing 
the film. In each case, the film group mean score was between 
one and two standard deviations higher than the mean score of 
the comparable control group that saw no film. 

2) The ‘sound irrelevant’ group had a lower mean score, in 
each population, than any other film group. It is to be noted, 
however, that the visual irrelevant version was the most effective 
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TABLE 2.—Raw MEAN Scores AND ADJUSTED MEANS or Navy 
AND ARMY POPULATIONS ON INFORMATION TEST AFTER VARIOUS 
Firm TREATMENTS 











Navy Army 
Treatment 

N Mean SD | N| Mean} SD 

Visual Relevant 136| 67.63 15.18} 248! 47.84 | 17.36 
(VR) (69.55) * (48.02) 

Visual Irrelevant | 148} 65.45 16.67 | 229} 49.83 | 17.26 
(VI) (66. 62) (49.24) 

Sound Relevant 135| 70.64 14.19| 226; 48.12 | 16.31 
(SR) (70.68) (49.00) 

Sound Irrelevant | 155] 66.67 15.17 | 242} 46.35 |17.71 
(SI) (65.11) (46.31) 

Basic (B) 331} 69.80 15.87 | 367} 52.55 | 18.80 
(69. 52) (52.09) 

Control (C) 150; 40.57 10.40 | 264} 30.04 9.74 
(39.88) (30.41) 























* Adjusted means are in parentheses. 


TABLE 3.—DIFFERENCES AMONG ADJUSTED MEAN SCORES FOR 
INFORMATION TEST? 


VR VI SR SI B 
Navy 
VI —2.93* —- — — am 
SR +1.13 +4.06** = — jie 
SI —4.44*** -—1.51 —5.57*** _ dan 
B — .03 +2.90* —1.16 +4.41*** — 
C -—29.67*** -—26.74*** -—30.80*** -—25.23*** —29.64*** 
Army 
VI +1.22 — — — _ 
SR + .98 — .24 oe — bonis 
SI —1.71 —2.93* —2.69* — _ 


B +4.07*** +2.85** +3.09** +5.77*** 
C —17.61*** —18.83*** -—18.59*** -—15.90*** -—21.68*** 
* In this table, the mean for the group listed on the top is subtracted from 
that of the group listed on the left. 
* Significant at the 5 per cent level of confidence. 
** Significant at the 1 per cent level of confidence. 
*** Significant at the 0.1 per cent level of confidence. 
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of all the versions containing attention devices for the Army 
population. 

3) The remainder of the inter-film differences were inconsistent 
for the two populations, and generally insignificant. In the 
Army, the basic version was consistently more effective than any 
device version, while in the Navy it was about equal to the two 
relevant device versions. 

4) No significant difference was noted as between the visual 
devices and the sound devices versions, combined without respect 
to relevance of devices. For the Navy, the weighted mean of 
the two visual devices versions was 68.02, as compared with a 
weighted mean of 67.70 for the sound devices versions. For 
the Army, the visual devices group mean, without respect to 
relevance of device, was 48.60, while the sound group mean was 
47.61. (These are all adjusted means.) 


Results of the Device Recognition Test 


The recognition test items were divided into two groups for 
each film version: (1) those devices that were in the film (Test C,) 
and (2) those devices that were not in the film (Test Cz). 

In scoring responses for the C, test, which comprised the 
devices actually in the film, a plus one (+1) was given for each 
device reported as present, a zero (0) if the subject reported he 
did not remember, and a minus one (—1) if the device was 
reported not present. The reverse of the scoring of the C, test 
was used for the C, test which comprised all devices not in the 
version being tested. A plus one (+1) was given for each device 
reported as not present, a zero (0) if the subject did not remember, 
and a minus one (—1) for each device reported as present. These 
scores were totaled for each film version and the tests (C, and 
C:) correlated with each other as well as with the film informa- 
tion test scores. Table 4 reports these correlations. 

All the correlations between the film information test and the 
C, test, and the film information test and the C, test, are 
generally insignificant although positive, indicating that there 
was only a slight relationship between scores on the film test 
and the remembering of the devices. The correlation between 
the C, and C; tests is also low. In all cases except one, which 
was practically zero, the correlations were negative. Analysis 
of the data showed that there was a slight tendency for the 
individual to mark the ‘not-present’ devices as present in the 








TABLE 4.—CORRELATIONS BETWEEN 
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DevicE RECOGNITION 


TresTs AND INFORMATION TrsT: C, Test-DEVICES IN THE VER- 
sIon, C2 Trest-Devices Not IN THE VERSION 














C, Test and | C2 Test and C, Test 
Information | Information and 
Group Test Test Cz Test 

Navy | Army | Navy | Army | Navy | Army 
Visual Relevant .185 | .191 | .191 | .830 |—.295|—.079 
Visual Irrelevant 308 | .299 | .182 | .344 042) .053 
Sound Relevant .223 | .092 | .141 | .203 |—.431|—.225 
Sound Irrelevant .107 | .075 | .073 | .150 |—.416)—.282 
Basic* .214 | .334 























*Since the Basic Version contained no devices there was no C, test score 
for this group. 


film, if he marked the ‘present’ devices as present. In other 
words, there was a general tendency to indicate that devices 
were present, whether they were used or not. 

An analysis of responses to the individual items included in 
the Device Recognition Test revealed two significant points. 
First, whereas only about half the items included in the VR, SR, 
and SI versions were recognized by seventy per cent or more of 
the groups seeing these versions, almost three quarters of the 
devices in the VI version were correctly recognized by seventy 
per cent or more of the groups seeing this version. In short, 
the visual irrelevant devices seemed to have called attention ‘to 
themselves’ to a greater extent than any other type of device did. 
Second, whereas a maximum of only seven devices were falsely 
recognized (i.e., they were not in the version but the subjects 
said they were) by forty per cent or more of the VR, VI, and SR 
version groups, fifteen items were falsely recognized by at least 
this proportion of the sound irrelevant group. It is suggested 
that the irrelevant sound devices tended to confuse the subjects 
so that they were uncertain about what they did hear. 


SUMMARY OF THE RESULTS 


The results may be summarized with respect to the three 
experimental hypotheses as follows: 
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Device relevancy.—There is no evidence that the insertion of 
relevant devices, as defined in this study, adds to the effective- 
ness of an informational film. There is some evidence that irrele- 
vant sound devices detract from the teaching effectiveness of 
such a film. (For both populations, the sound irrelevant ver- 
sion yielded the lowest scores.) To a lesser degree there is 
evidence that any of the devices used may have the same effect. 
(For the Army populations the basic version—no devices— 
yielded the highest score.) 

Device medium.—No significant difference was found between 
the effectiveness (or ineffectiveness) of visual and sound atten- 
tion-gaining devices. 

Device recognition.—Ability to recognize and remember which 
devices were used in a film version was practically independent 
of learning from the version. 


CONCLUSIONS AND RECOMMENDATIONS 


This study leads to the general conclusion that attention- 
gaining devices of the kinds inserted in these films do not add 
significantly to learning. 

The following recommendations are suggested as a result of 
this investigation: 

1) Where instruction is the principal aim, and cost a con- 
sideration, producers of training films should present the subject 
matter in a simple, straightforward way and avoid the use of such 
fancy and expensive devices as spotlighting, zooms, extreme 
magnification, and stop motion, to gain the learners’ attention. 

2) If it seems necessary, in an instructional film, to use devices 
to attract or direct the learners’ attention, use a technique which 
will emphasize something already in the film—some special 
treatment of indigenous materials related to the subject content 
—rather than introduce extraneous or irrelevant materials. 

The findings of this experiment are largely negative; however, 
that does not nullify the usefulness of the study. It is almost 
as important to discover that certain techniques are ineffective, 
as it is to learn that other techniques are effective. In general, 
this study corroborates the belief which many educators have 
held for a long time, that it is not necessary to have a fancy, 
expensive treatment to convey ideas by means of film, and that a 
simple straightforward treatment is often the most effective. 











A COMPARISON OF READING ABILITY AND 
READABILITY 
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Numerous attempts have been made to develop a satisfactory 
technique for determining the level of reading difficulty of 
printed materials. There are three of these which are given 
most extensive use at the present time; namely, those proposed 
by Lorge, Flesch, and Dale and Chall. Since their appearance 
many types of materials have been appraised by one or more of 
the formulae: textbooks and children’s books; advertising;! 
industrial publications;'* pamphlets vn international relation- 
ships;'° newspaper materials;'* corporate annual reports.?! 
The formulae have been used by the American Library Associa- 
tion* and the United States Department of Agriculture Exten- 
sion Service.*? 

Since these techniques essentially provide a type of ‘measur- 
ing’ instrument, they should be examined to determine their 
validity and reliability. The first study on reliability, i.e., 
analyst-to-analyst reliability, of the Flesch formula was reported 
by Haynes, Jenkins and Walker.'? They conclude that “For 
practical purposes the Flesch formulas and the directions for 
their use are sufficiently objective to be used even by inexpe- 
rienced analysts to obtain estimates of reading ease and human 
interest of written materials.’”’ As for validity, Flesch found 
that materials rated as more readable by his formula were also 
more readable according to readership surveys and to expert 
opinion. The studies which the present writers have located 
have relied upon this latter type of criterion of readability. 
The subjective nature of such a criterion raises the question of 
its reliability. 

PURPOSE 


For the present paper it is assumed that materials rated as 
readable at a certain grade level should be understandable to 
students at that level. Thus, if a readability formula indicates 
that certain material is located at the Grade IX level, then it 
is to be expected that children at the Grade IX reading level 
491 
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should be able to read the material and then to answer questions 
on it with a high degree of accuracy. 

The purpose of the present paper is twofold: (a) to determine 
the readability of certain United Nations publications by each 
of the three readability formulae mentioned above; (b) to deter- 
mine the reading ability of a group of Grade XII students by 
means of some standardized reading test, and the accuracy with 
which these same students can answer questions based on the 
United Nations materials. 

The need for such a study has been emphasized in numerous 
papers appearing during the last few years. Lostutter'® con- 
cluded that the Flesch formula was best adapted for the measure- 
ment of the readability of newsprint. Lorge’’ has stated that 
each of the three formulae has a specific function to perform. 
He claims that his formula will give “‘evidence on the com- 
prehension difficulty of the kind of reading pupils have to do in 
school,’”’ that the Flesch formula provides a measure of read- 
ability of magazine materials, and that the Dale-Chall formula 
“‘will make for. adequate appraisal of health materials.’’ Flesch, 
on the other hand, has pointed out that, ‘‘ There is no theoretical 
justification for this neat pigeonholing of formulas; all three of 
those mentioned were based on exactly the same criterion; 
namely, the McCall-Crabs’ Standard Test Lessons in Reading.” 
Both Lorge and Flesch agree, however, that there is some degree 
of correlation between the formulae when applied to the same 
materials: ‘‘The statistical differences between their predictive 
values are insignificant.”” In a recent paper Michaelis’ indi- 
cated that some U. N. materials were written on a difficult level 
according to teachers’ judgments and data from readability 
formulae. He reported also that there was considerable disparity 
between the difficulty levels obtained by the different formulae. 


TEST MATERIALS 


The Iowa Silent Reading Test (Form Cm) was administered to 
forty-two boys and thirty-six girls in Grade XII at the Oakland 
Technical High School to determine the grade level at which 
these students were reading. The same students were also 
given an objective test based upon several U. N. publications 
which had been prepared for use in high-school social studies 


classes. 
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The U. N. test included four subtests: vocabulary, sentence 
comprehension, paragraph comprehension, and reading for 
details. The first part required the students to read a paragraph 
chosen from one of the U. N. pamphlets. The students were 
allowed three minutes to read and study the materials. At the 
end of that time the printed materials were collected and the 
students attempted to answer questions based on their reading. 
There were nine multiple-choice questions in Part I. The second 
part was a vocabulary test, again multiple-choice in nature. 
The test included twenty-two words and phrases, such as “‘pre- 
amble,” ‘‘scope,” ‘‘indigenous,” “the framers of the Constitu- 
sion,” and “‘ocean rendezvous.”’ The third part consisted of a 
series of sentences, a phrase in each sentence being underlined. 
The students were required to indicate which of four choices 
could most reasonably be substituted for the underlined expres- 
tion. The last part attempted to measure the students’ under- 
standing of short paragraphs. The multiple-choice questions 
were arranged to test the ability of the student to select the best 
title for the paragraph; to test his knowledge of word meanings; 
to test his ability to read for details. There was also a free 
expression question on each paragraph, but this item is not used 
in the present report. The reliability of the test of sixty-nine 
items, determined by the split-half method and stepped-up by 
the Spearman-Brown Prophecy formula, was .85. 


FINDINGS 


The readability of the U. N. materials was determined by 
each of the formula listed above. The Lorge formula considers 
average sentence length, ratio of prepositional phrases to number 
of words, and ratio of hard words to number of words in various 
samples of the materials being appraised. The Flesch formula 
for reading ease involves the determination of number of syllables 
per one hundred words and average sentence length. The 
Dale-Chall formula requires the calculation of average sentence 
length and the number of hard words. Readability by each 
formula was determined on the same sampling of U. N. publica- 
tions. Five selections were chosen from each of five pamphlets 
according to the directions for the application of the formulae. 
Similarly the proper number of words was included in each 
sample. The computation of readability was done on two separate 
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occasions with practically identical results. Furthermore, five 
other samples were chosen from each of the five pamphlets. 
Readability was determined for each of these samples, the 
results checking with those obtained from the previous sampling. 
Also the U. N. test was prepared in the main from the materials 
that had been used to determine readability. The readability 
of the test materials and of the instructions for the test was also 
determined by each formula. The U. N. publications varied 
from seven to two hundred one pages in length and were believed 
to be representative of those available for use in the public 
schools. 


TABLE I.—READABILITY OF UNITED NATIONS MATERIALS 
APPRAISED BY THREE FORMULAE 





























Ratings According to Formulae 
United Nations Dale-Chall Flesch Lorge 
Material 
Raw Grade Raw | Levelof | Raw Grade 
Score | “°° | Score Difficulty | Score Place- 
ment ment 
Basic Facts 9.6226) 13-15 |28.6720| College 8.6524) 8+ 
graduate 
Everyman’s UNO 9.4727| 13-15 |24.2233/ College (8.3557) 8+ 
graduate 
FAO 8.2895} 11-12 |38.9930) 13-16 7.5497], 7+ 
Guide to UN Charter /9.8611) 13-15 {31.7220} 13-16 8.1380) 8+ 
How Peoples Work |9.5917| 13-15 |37.6400) 13-16 7.9803} 8 
Together 
Average 9.3675) 13-15 |32.2500) 13-16 8.1352) 8+ 








Table I shows the readability of the five pamphlets computed 
by each of the three formulae discussed above. From the table 
it may be seen that the Lorge formula indicates that the materials 
should be readable at about the Grade VIII level. According 
to the Flesch formula the materials are rated at the college under- 
graduate and graduate levels. Readability as estimated by the 
Dale-Chall method is placed at Grades XI to XV for the different 
pamphlets. There is, then, considerable discrepancy between 


the results obtained from different readability formulae. 
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As has been mentioned, it has been assumed for this study 
that if materials are readable at a certain level, then students 
at that level of reading ability should be able to read the mate- 
rials with a high degree of accuracy. Specifically this would 
mean that according to the Lorge formula children who read 
at about the Grade VIII level should be able to comprehend the 
U. N. materials. Table II shows the mean percentile rank on 
the Iowa Silent Reading Test of the seventy-eight subjects in 
Grade XII according to the manual of directions. 


TABLE II.—MEAN STANDARD SCORES AND PERCENTILE RANKS 
oF SUBJECTS 


Mean Standard Percentile Ranks 


Subtest Scores in Grade XII 
1. Rate 171 41 
1. Comprehension 203 92 
2. Directed Reading 166 42 
3. Poetry Comprehension 172 50 
4. Word Meaning 179 56 
5. Sentence Meaning 171 35 
6. Paragraph Comprehension 176 58 
7. Use of Index 177 63 
7. Selection of Key Words 169 43 


The mean standard score on the whole test was 174, which 
corresponds to a grade equivalent of 13.0, indicating that these 
students were on the average reading at about the Grade XIII 
level. 

Before indicating the success of these subjects on the U. N. 
reading test, it is desirable to indicate the readability level of the 
test materials and of the instructions to the test. This is done 
in Table III. 

The sampling of test materials was no more difficult than was 
the sampling of the U. N. publications; the instructions were 
readable at a slightly lower level than were the test materials. 
The U. N. test was slightly easier than were the pamphlets 
upon which it had been based because the readability score for 
the test included the estimate of the readability of the instruc- 
tions. It now remains to indicate how well the students were 
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TaBLeE IIJ.—Reapapiuiry or U. N. Test anp or TEst 
INSTRUCTIONS 





U. N. Test Test Instructions 





Raw Raw Inter- 


Formula Interpretati 
Score eet ieee Score pretation 





Dale-Chall 8.8625 | Grades 11-12 6.7622 | Grades 7-8 
Flesch 45.3711 | Grades 13-16 | 64.8980 | Grades 8-9 
Lorge 7.7462 | Grades 7+ 6.3335 | Grades 6+ 

















TaBLE IV.—MEAN Scores on U. N. REaApinG TEst 
Mean Percentage 


Subtests No. of Items Mean Correct 
l 9 6.44 71.56 
2 22 15.47 70.32 
3 6 3.79 63.17 
4 32 17.36 54.25 
Total 69 43.06 62.41 


able to perform on the U. N. reading test. Mean scores on each 
of the subtests are shown in Table IV. 

In the absence of any absolute standard of how well students 
reading at the Grade XIII level should be able to perform on 
the U. N. reading test, it still appears reasonable to conclude 
that even at this level of ability the test offers considerable 
difficulty. It would seem that students who are reading below 
the Grade XIII level would experience even more trouble. 
Judged by this criterion of reading ability, the readability 
formulae appear to place the U. N. materials at too easy a level, 
some formulae showing a greater disparity between reading 
ability and readability than do others. 


SUMMARY AND CONCLUSIONS 


The present findings offer support to Lorge’s contention that 
different formulae may be more suitable for some materials 
than for others. If this is so, there is need for research to deter- 
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mine the most appropriate formula to use with specific types of 
materials. There seems no doubt that factors other than those 
now being considered must be employed if suitable formulae 
are to be developed. As Lorge has pointed out, present tech- 
niques really do not assess the difficulty of the concepts found 
in the materials. Length of word and number of syllables are 
not necessarily correlated with abstractness of a concept, for 
some very abstract concepts involve few letters, whereas some 
concrete concepts are expressed by lengthy words. 

The following conclusions appear to be justified by the evidence 
here presented: 

1) There is considerable disparity between the readability 
levels found for U. N. materials by different readability formulae. 

2) The Flesch and the Dale-Chall formulae estimate the read- 
ability of this type of material somewhat more accurately than 
does the Lorge formula, if level of reading ability is used as the 
criterion. 

3) The U. N. materials sampled appear to be too difficult for 
high-school students. 

Finally, it would appear that there is need for extended study 
of the readability of various types of materials intended for use 
in the public schools. These studies might well be carried out 
on regular types of school materials as well as on special materials 
such as pamphlets at various grade levels. Further study of 
the problem of readability is indicated, probably especially 
designed to consider the nature of the concepts involved. Lastly, 
it may be that the basic formulae are logically sound, and that 
the problem is the development of more valid norms. 
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NOTES ON A SUGGESTED INDEX OF ITEM 
VALIDITY: THE U-L INDEX 


A. PEMBERTON JOHNSON 
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Princeton, New Jersey 


In 1935 Long, Sandiford, et al,* reported an empirical com- 
parative study of the effectiveness of thirteen validity techniques 
in selecting from among four hundred eighty-two six-choice 
vocabulary items that set of fifty and that set of one hundred 
items which correlated highest with criterion scores based on all 
four hundred eighty-two items. For this situation they con- 
cluded, among other things, that those validity techniques which 
tend to discriminate in favor of items of fifty per cent difficulty 
appear to be superior to the techniques such as bi-serial r, Kelley 
and Vincent which make no discrimination in favor of median 
difficulty items. Considering both effectiveness and ease of 
computation, they concluded that among those techniques 
tending to select items of fifty per cent difficulty (percentage 
passing), the upper vs. lower (group) technique was to be pre- 
ferred. They stated that item validities derived by the upper 
vs. lower technique tend to reach maximum reliability when the 
proportion of cases in each of these groups is twenty-seven per 
cent.t Numerically the item validities were expressed in this 
technique by the difference between number of correct responses 
for the highest twenty-seven per cent (or by the highest third) 
and the number of correct responses for the lowest twenty-seven 
per cent (or the lowest third) of the cases as arrayed on the cri- 
terion scale. 

This paper proposes the U-L Index, a simple modification of 
the upper-lower twenty-seven per cent technique. This index 
proved extremely useful during World War II as a basis for 
selecting items to be included in Army Air Forces bombardier 
knowledge tests. The U-L Index effectively tends to select 





*John A. Long, Peter Sandiford, and Others. The Validation of Test 
Items. Bulletin No. 3, Dept. of Ed. Res., University of Toronto, Toronto, 
1935, pp. 94-5. 

t See also T. L. Kelley, ‘‘The Selection of Upper and Lower Groups for the 
Validation of Test Items,” J. Educ. Psychol., xxx, January, 1939, pp. 17-24, 
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items of moderate difficulty which show good separation betwéen 
upper and lower twenty-seven per cent groups. It is more 
easily computed and interpreted than any conventional item 
validity index known to the author. For many practical test 
construction situations where expensive item analysis cannot be 
justified, it is believed to be quite useful. 

This modification possibly has been used by other investi- 
gators before, but is believed not to have been. It requires the 
difference between the groups in number of correct responses to 
be divided by the number of cases in each group. Symbolically, 
if 

R, = number correct responses in the upper group of 
twenty-seven per cent 

R, = number correct responses in the lower group of 
twenty-seven per cent 

N = number of cases 


f =.27N 
_Rku-h_ RR 
U-L Index = —oIN = i (1) 


This index can theoretically vary from +1 through 0 to —1. 
These proportions are independent of the N of the population 
sample and similar in magnitude though not strictly in meaning 
to conventional correlation coefficients. 

Long, Sandiford, et al., found in their empirical study that (1) 
the average difficulty of a group of items shown to be least valid 
by the bi-serial r technique was about equal to the average 
difficulty of the group of items indicated to be least valid by the 
upper-lower thirds technique, but (2) that the bi-serial r tended 
to eliminate items indiscriminately throughout the entire range 
of item difficulty whereas the upper-lower thirds technique 
tended to eliminate items farthest removed from the fifty per 
cent level of difficulty (percentage passing).* 

In a table of intercorrelations among the validity values 
obtained for one hundred items by the thirteen methods which 
they studied, Long, Sandiford, et al., report a correlation of 
.65 between bi-serial r and upper-lower thirds validities. This 
finding strongly suggests the need to interpret the proposed 





* Ibid., p. 114. 
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U-L Index in somewhat different fashion than the conventional 
bi-serial r. 

The standard error of the U-L Index may be derived from the 
conventional formula for the standard error of the difference of 
two proportions, when the data are uncorrelated. 

Guilford* gives the following generalized formulae: 


#34 Cap = Von, + op,” — 2rop,ons 


#28 c= a 
where: og, = S.E. of a difference between two proportions 

o, = S.E. of a proportion 

Pi = lst proportion 

P2 = 2nd proportion 

q=i-p 

N = number of cases 

r = coefficient of intercorrelation of the two sets of 
data. 

Using the previously mentioned symbols for the U-L Index: 


—— = JG) (: - *) a, A R.) 


N f 
a 


2 
Similarly hin AL we - 























In Guilford’s formula #34 for cap, the most conservative esti- 
mate of ris0. That formula then reduces to cap = +/o5,? + ops" 
By substitution: 


1 R,? 
SE U-L Index =F fRe - F +R, - 








R? 
f 
ia R,? + R;? 


The formula for the standard error of the U-L Index is such 
that the maximum value of the standard error occurs in the 











*J. P. Guilford. Fundamental Statistics in Psychology and Education, 
New York: McGraw-Hill, 1942, pp. 133 and 142. 








502 The Journal of Educational Psychology 


TaBLE I.—5% anv 1% “t’’ Test SIGNtIFIcANcE LEVELS FOR 
THE SE or U-L INDEX WHEN THE U-L INDEx = 0 anp 50% 
Pass THE ITEM* 


N f= .27N SEof the U-L Index 5%Level 1% Level 


100 27 . 136 . 273 . 364 
148 40 .112 . 223 . 295 
200 54 .096 . 190 . 252 
248 67 .087 .172 . 227 
300 81 .079 . 156 . 206 
348 94 .073 .144 . 190 
400 108 .068 . 134 .177 
500 135 .061 . 120 . 159 
1000 270 .043 . 084 111 


* For example, only 1 per cent of obtained values would be expected to 
exceed +.364 were the true index zero for a sample of one hundred cases. 
In this table the degrees of freedom have been taken as equal to 2(f-1) 
since f is defined as the number in each of the two contrasted groups. 


fifty per cent passing situation where the true index equals zero. 
Table I below gives the 5% and 1% “‘t”’ significance levels of 
the SE of the U-L Index in that situation. 

A quite serviceable index of item difficulty (percentage passing) 
for each item can be obtained from the values for R, and R, by 
use of the following formula: 

Difficulty level (percentage passing) = 100 - Ri) 
_ 50(Ru + Ri) 
f 





(3) 





Since this is based on only the extreme fifty-four per cent of 
the cases, it does not give exactly the value for the index of item 
difficulty which would be obtained from all of the cases. In 
practice however the discrepancy usually is not appreciable. 

Formula (1) for the U-L Index provides no specific correction 
for omissions, although it is necessary sometimes to item analyze 
speeded tests in which the number of omissions among later 
items is appreciable. 

Omissions on speeded tests are usually of two types: (a) those 
by persons who read the item but fail to mark any choice and 
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(b) those by persons who have not read as far in the test as the 
end of that item. Davis* in a communication to the writer 
suggests a method of determining the ‘non-reads’ directly. The 
test papers are scored and arranged in rank order according to 
the criterion score. The group which ineludes the top twenty- 
seven per cent of papers is segregated as is the group which com- 
prises the bottom twenty-seven per cent of test papers. Next 
the papers of the upper group are gone through to determine 
which item on each paper is the last one marked. A tally is 
then made opposite the number of the next item for that one is 
presumably the first item not read by the subject. This tally 
is the basis for a cumulative frequency table of the number of 
cases within the upper group who have not read each particular 
item. A similar count is made of the ‘non-reads’ in the lower 
group or the bottom twenty-seven per cent of the test papers. 

By graphic item count on the International Test Scoring 
machine, or by other means, the number of right responses and 
the number of wrong responses in the upper group to each item 
isfound. The omits may be obtained, if desired, by subtracting 
from the number of papers within the upper group the rights 
plus wrongs plus ‘non-reads.’ Similarly the omits among the 
lower group may be obtained, if desired. 

Where the ‘non-reads’ represent an appreciable proportion of 
either criterion group the following formula for the U-L Index 
is suggested: 


Ry = R: 
(f — ‘non-reads’,) (f — ‘non-reads’;) 





(4) 


where R, = number of right responses in the upper group 
R; = number of right responses in the lower group 
f = .27N 
‘non-reads’, = ‘non-reads’ in the upper group 
‘non-reads’; = ‘non-reads’ in the lower group 
Ry 
‘non-reads’,) 
responses in the upper group to the number in that group who 





The proportion a is the proportion of right 





* Dr. Frederick B. Davis, formerly with the Codperative Test Service 
and the Aviation Psychology Program of the Army Air Forces, now at Hun- 
ter College. 
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ma 
‘non-reads’;) y be 


read the item. The proportion F— 
similarly interpreted for the lower group. 

The U-L Index is proposed as a useful means of selecting items 
either to increase the internal consistency (odd-even reliability) 
of a test or to increase its correlation with a given criterion. 
Some advantages of the U-L Index are: (1) the demonstrated 
effectiveness of the upper vs. lower twenty-seven per cent 
technique on which it is based, (2) its ease of computation, and 
(3) the readiness with which its level of significance may be 


approximated. 
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